Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: checker handle source unavailable #1134

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

AleksandrMatsko
Copy link
Member

@AleksandrMatsko AleksandrMatsko commented Jan 14, 2025

PR Summary

In #1085 retries to Graphite remote metrics source were added. So new error ErrRemoteUnavailable was added.

This PR refactors checker logic to proper handle ErrRemoteUnavailable and ErrRemoteTriggerResponse. Also add tests for handling fetch errors

@AleksandrMatsko AleksandrMatsko changed the title Fix/checker handle src unavailable fix/checker handle source unavailable Jan 14, 2025
@AleksandrMatsko AleksandrMatsko changed the title fix/checker handle source unavailable fix: checker handle source unavailable Jan 14, 2025
@@ -132,24 +132,17 @@ func (triggerChecker *TriggerChecker) handleFetchError(checkData moira.CheckData
triggerChecker.trigger.ClusterKey(),
)
}
case remote.ErrRemoteTriggerResponse:
timeSinceLastSuccessfulCheck := checkData.Timestamp - checkData.LastSuccessfulCheckTimestamp
if timeSinceLastSuccessfulCheck >= triggerChecker.ttl {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Убрал это условие, т.к. оно было очень странным:

  • если "повезло" и источник оказался недоступен в момент проверки триггера, то триггер перейдёт в EXCEPTION, если с момента проверки прошло более ttl секунд (это которые "Set NODATA if has no value for ttl seconds"). Плюс к этому здесь генерилось 2 NotificationEvent, в случае выполнения условия.
  • иначе триггер состояние не поменяет + к этому, если источник остаётся не доступным, то триггер вообще не будет проверяться (т.к. если источник недоступен, то триггеры не шедулятся на проверку), т.е. не перейдёт в EXCEPTION

Если я что-то не учёл/не правильно понял, сигнализируйте)

@AleksandrMatsko AleksandrMatsko marked this pull request as ready for review January 15, 2025 07:43
@AleksandrMatsko AleksandrMatsko requested a review from a team as a code owner January 15, 2025 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant